08/09/2024 - 14/09/2024

10/09/2024 19:35

This output seems to verify we are indeed reading and writing to the device

[root@fe01 tests]# ./run_test.sh
Info: Number of enabled h2c channels = 2
Info: Number of enabled c2h channels = 2
Info: The PCIe DMA core is memory mapped.
Info: Running PCIe DMA memory mapped write read test
      transfer size:  1024
      transfer count: 1
Info: Writing to h2c channel 0 at address offset 0.
Info: Writing to h2c channel 1 at address offset 1024.
Info: Wait for current transactions to complete.
sscanf() = 1, value = 0x00000400
sscanf() = 1, value = 0x00000000
sscanf() = 1, value = 0x00000001
device = /dev/xdma0_h2c_0, address = 0x00000000, size = 0x00000400, offset = 0x00000000, count = 1
Host memory buffer = 0x196f400
Data read from file into buffer at 0x196f400 (size: 1024 bytes):
10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f 20 21 22 23 24 25 26 27 28 29 2a 2b 2c 2d 2e 2f
sscanf() = 1, value = 0x00000400
sscanf() = 1, value = 0x00000400
sscanf() = 1, value = 0x00000001
Data written to FPGA device at 0x196f400 (iteration 0):
device = /dev/xdma0_h2c_1, address = 0x00000400, size = 0x00000400, offset = 0x00000000, count = 1
10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f 20 21 22 23 24 25 26 27 28 29 2a 2b 2c 2d 2e 2f
Host memory buffer = 0xcf1400
CLOCK_MONOTONIC reports 0.000065215 seconds (total) for last transfer of 1024 bytes
Data read from file into buffer at 0xcf1400 (size: 1024 bytes):
Transfer speed: 14.97 MB/s
00 00 00 00 01 00 00 00 02 00 00 00 03 00 00 00 04 00 00 00 05 00 00 00 06 00 00 00 07 00 00 00
Data written to FPGA device at 0xcf1400 (iteration 0):
00 00 00 00 01 00 00 00 02 00 00 00 03 00 00 00 04 00 00 00 05 00 00 00 06 00 00 00 07 00 00 00
CLOCK_MONOTONIC reports 0.000054443 seconds (total) for last transfer of 1024 bytes
Transfer speed: 17.94 MB/s
Info: Writing to h2c channel 0 at address offset 2048.
Info: Writing to h2c channel 1 at address offset 3072.
Info: Wait for current transactions to complete.
sscanf() = 1, value = 0x00000400
sscanf() = 1, value = 0x00000800
sscanf() = 1, value = 0x00000001
device = /dev/xdma0_h2c_0, address = 0x00000800, size = 0x00000400, offset = 0x00000000, count = 1
Host memory buffer = 0x7e4400
Data read from file into buffer at 0x7e4400 (size: 1024 bytes):
ff ff 00 00 ff fe 00 01 ff fd 00 02 ff fc 00 03 ff fb 00 04 ff fa 00 05 ff f9 00 06 ff f8 00 07
Data written to FPGA device at 0x7e4400 (iteration 0):
ff ff 00 00 ff fe 00 01 ff fd 00 02 ff fc 00 03 ff fb 00 04 ff fa 00 05 ff f9 00 06 ff f8 00 07
CLOCK_MONOTONIC reports 0.000058135 seconds (total) for last transfer of 1024 bytes
Transfer speed: 16.80 MB/s
sscanf() = 1, value = 0x00000400
sscanf() = 1, value = 0x00000c00
sscanf() = 1, value = 0x00000001
device = /dev/xdma0_h2c_1, address = 0x00000c00, size = 0x00000400, offset = 0x00000000, count = 1
Host memory buffer = 0x135b400
Data read from file into buffer at 0x135b400 (size: 1024 bytes):
00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f
Data written to FPGA device at 0x135b400 (iteration 0):
00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f
CLOCK_MONOTONIC reports 0.000040952 seconds (total) for last transfer of 1024 bytes
Transfer speed: 23.85 MB/s
Info: Reading from c2h channel 0 at address offset 0.
sscanf() = 1, value = 0x00000400
sscanf() = 1, value = 0x00000000
sscanf() = 1, value = 0x00000001
device = /dev/xdma0_c2h_0, address = 0x00000000, size = 0x00000400, offset = 0x00000000, count = 1
Host memory buffer = 0x1f3c000
Info: Reading from c2h channel 1 at address offset 1024.
Data read into buffer at 0x1f3c000 (iteration 0):
10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f 20 21 22 23 24 25 26 27 28 29 2a 2b 2c 2d 2e 2f
Info: Wait for the current transactions to complete.
sscanf() = 1, value = 0x00000400
sscanf() = 1, value = 0x00000400
sscanf() = 1, value = 0x00000001
device = /dev/xdma0_c2h_1, address = 0x00000400, size = 0x00000400, offset = 0x00000000, count = 1
Host memory buffer = 0x1fcb000
Data read into buffer at 0x1fcb000 (iteration 0):
00 00 00 00 01 00 00 00 02 00 00 00 03 00 00 00 04 00 00 00 05 00 00 00 06 00 00 00 07 00 00 00
CLOCK_MONOTONIC reports 0.000024453 seconds (total) for last transfer of 1024 bytes
Transfer speed: 39.94 MB/s
CLOCK_MONOTONIC reports 0.000021195 seconds (total) for last transfer of 1024 bytes
Transfer speed: 46.08 MB/s
Info: Reading from c2h channel 0 at address offset 2048.
sscanf() = 1, value = 0x00000400
sscanf() = 1, value = 0x00000800
sscanf() = 1, value = 0x00000001
device = /dev/xdma0_c2h_0, address = 0x00000800, size = 0x00000400, offset = 0x00000000, count = 1
Host memory buffer = 0x1c62000
Info: Reading from c2h channel 1 at address offset 3072.
Data read into buffer at 0x1c62000 (iteration 0):
ff ff 00 00 ff fe 00 01 ff fd 00 02 ff fc 00 03 ff fb 00 04 ff fa 00 05 ff f9 00 06 ff f8 00 07
Info: Wait for the current transactions to complete.
sscanf() = 1, value = 0x00000400
sscanf() = 1, value = 0x00000c00
sscanf() = 1, value = 0x00000001
device = /dev/xdma0_c2h_1, address = 0x00000c00, size = 0x00000400, offset = 0x00000000, count = 1
Host memory buffer = 0x6d6000
Data read into buffer at 0x6d6000 (iteration 0):
00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f
CLOCK_MONOTONIC reports 0.000025026 seconds (total) for last transfer of 1024 bytes
Transfer speed: 39.02 MB/s
CLOCK_MONOTONIC reports 0.000022440 seconds (total) for last transfer of 1024 bytes
Transfer speed: 43.52 MB/s
Info: Checking data integrity.
Info: Data check passed for address range 0 - 1024.
Info: Data check passed for address range 1024 - 2048.
Info: Data check passed for address range 2048 - 3072.
Info: Data check passed for address range 3072 - 4096.
Info: All PCIe DMA memory mapped tests passed.
Info: All tests in run_tests.sh passed.
[root@fe01 tests]#
[root@fe01 tests]# ./run_test.sh
Info: Number of enabled h2c channels = 2
Info: Number of enabled c2h channels = 2
Info: The PCIe DMA core is memory mapped.
Info: Running PCIe DMA memory mapped write read test
      transfer size:  1024
      transfer count: 1
Info: Writing to h2c channel 0 at address offset 0.
Info: Writing to h2c channel 1 at address offset 1024.
Info: Wait for current transactions to complete.
sscanf() = 1, value = 0x00000400
sscanf() = 1, value = 0x00000000
sscanf() = 1, value = 0x00000001
device = /dev/xdma0_h2c_0, address = 0x00000000, size = 0x00000400, offset = 0x00000000, count = 1
Host memory buffer = 0x196f400
Data read from file into buffer at 0x196f400 (size: 1024 bytes):
10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f 20 21 22 23 24 25 26 27 28 29 2a 2b 2c 2d 2e 2f
sscanf() = 1, value = 0x00000400
sscanf() = 1, value = 0x00000400
sscanf() = 1, value = 0x00000001
Data written to FPGA device at 0x196f400 (iteration 0):
device = /dev/xdma0_h2c_1, address = 0x00000400, size = 0x00000400, offset = 0x00000000, count = 1
10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f 20 21 22 23 24 25 26 27 28 29 2a 2b 2c 2d 2e 2f
Host memory buffer = 0xcf1400
CLOCK_MONOTONIC reports 0.000065215 seconds (total) for last transfer of 1024 bytes
Data read from file into buffer at 0xcf1400 (size: 1024 bytes):
Transfer speed: 14.97 MB/s
00 00 00 00 01 00 00 00 02 00 00 00 03 00 00 00 04 00 00 00 05 00 00 00 06 00 00 00 07 00 00 00
Data written to FPGA device at 0xcf1400 (iteration 0):
00 00 00 00 01 00 00 00 02 00 00 00 03 00 00 00 04 00 00 00 05 00 00 00 06 00 00 00 07 00 00 00
CLOCK_MONOTONIC reports 0.000054443 seconds (total) for last transfer of 1024 bytes
Transfer speed: 17.94 MB/s
Info: Writing to h2c channel 0 at address offset 2048.
Info: Writing to h2c channel 1 at address offset 3072.
Info: Wait for current transactions to complete.
sscanf() = 1, value = 0x00000400
sscanf() = 1, value = 0x00000800
sscanf() = 1, value = 0x00000001
device = /dev/xdma0_h2c_0, address = 0x00000800, size = 0x00000400, offset = 0x00000000, count = 1
Host memory buffer = 0x7e4400
Data read from file into buffer at 0x7e4400 (size: 1024 bytes):
ff ff 00 00 ff fe 00 01 ff fd 00 02 ff fc 00 03 ff fb 00 04 ff fa 00 05 ff f9 00 06 ff f8 00 07
Data written to FPGA device at 0x7e4400 (iteration 0):
ff ff 00 00 ff fe 00 01 ff fd 00 02 ff fc 00 03 ff fb 00 04 ff fa 00 05 ff f9 00 06 ff f8 00 07
CLOCK_MONOTONIC reports 0.000058135 seconds (total) for last transfer of 1024 bytes
Transfer speed: 16.80 MB/s
sscanf() = 1, value = 0x00000400
sscanf() = 1, value = 0x00000c00
sscanf() = 1, value = 0x00000001
device = /dev/xdma0_h2c_1, address = 0x00000c00, size = 0x00000400, offset = 0x00000000, count = 1
Host memory buffer = 0x135b400
Data read from file into buffer at 0x135b400 (size: 1024 bytes):
00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f
Data written to FPGA device at 0x135b400 (iteration 0):
00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f
CLOCK_MONOTONIC reports 0.000040952 seconds (total) for last transfer of 1024 bytes
Transfer speed: 23.85 MB/s
Info: Reading from c2h channel 0 at address offset 0.
sscanf() = 1, value = 0x00000400
sscanf() = 1, value = 0x00000000
sscanf() = 1, value = 0x00000001
device = /dev/xdma0_c2h_0, address = 0x00000000, size = 0x00000400, offset = 0x00000000, count = 1
Host memory buffer = 0x1f3c000
Info: Reading from c2h channel 1 at address offset 1024.
Data read into buffer at 0x1f3c000 (iteration 0):
10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f 20 21 22 23 24 25 26 27 28 29 2a 2b 2c 2d 2e 2f
Info: Wait for the current transactions to complete.
sscanf() = 1, value = 0x00000400
sscanf() = 1, value = 0x00000400
sscanf() = 1, value = 0x00000001
device = /dev/xdma0_c2h_1, address = 0x00000400, size = 0x00000400, offset = 0x00000000, count = 1
Host memory buffer = 0x1fcb000
Data read into buffer at 0x1fcb000 (iteration 0):
00 00 00 00 01 00 00 00 02 00 00 00 03 00 00 00 04 00 00 00 05 00 00 00 06 00 00 00 07 00 00 00
CLOCK_MONOTONIC reports 0.000024453 seconds (total) for last transfer of 1024 bytes
Transfer speed: 39.94 MB/s
CLOCK_MONOTONIC reports 0.000021195 seconds (total) for last transfer of 1024 bytes
Transfer speed: 46.08 MB/s
Info: Reading from c2h channel 0 at address offset 2048.
sscanf() = 1, value = 0x00000400
sscanf() = 1, value = 0x00000800
sscanf() = 1, value = 0x00000001
device = /dev/xdma0_c2h_0, address = 0x00000800, size = 0x00000400, offset = 0x00000000, count = 1
Host memory buffer = 0x1c62000
Info: Reading from c2h channel 1 at address offset 3072.
Data read into buffer at 0x1c62000 (iteration 0):
ff ff 00 00 ff fe 00 01 ff fd 00 02 ff fc 00 03 ff fb 00 04 ff fa 00 05 ff f9 00 06 ff f8 00 07
Info: Wait for the current transactions to complete.
sscanf() = 1, value = 0x00000400
sscanf() = 1, value = 0x00000c00
sscanf() = 1, value = 0x00000001
device = /dev/xdma0_c2h_1, address = 0x00000c00, size = 0x00000400, offset = 0x00000000, count = 1
Host memory buffer = 0x6d6000
Data read into buffer at 0x6d6000 (iteration 0):
00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f
CLOCK_MONOTONIC reports 0.000025026 seconds (total) for last transfer of 1024 bytes
Transfer speed: 39.02 MB/s
CLOCK_MONOTONIC reports 0.000022440 seconds (total) for last transfer of 1024 bytes
Transfer speed: 43.52 MB/s
Info: Checking data integrity.
Info: Data check passed for address range 0 - 1024.
Info: Data check passed for address range 1024 - 2048.
Info: Data check passed for address range 2048 - 3072.
Info: Data check passed for address range 3072 - 4096.
Info: All PCIe DMA memory mapped tests passed.
Info: All tests in run_tests.sh passed.
[root@fe01 tests]#

and the output of the script I wrote seems to verify we are indeed reading the same thing:

[root@fe01 bin]# ./PCIe_DMA_Readout /dev/xdma0_c2h_0 0x00000000 16
Transfer speed: 0.11 MB/s
0000: 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f  |................|
[root@fe01 bin]# ./PCIe_DMA_Readout /dev/xdma0_c2h_0 0x00000000 32
Transfer speed: 1.24 MB/s
0000: 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f  |................|
0010: 20 21 22 23 24 25 26 27 28 29 2a 2b 2c 2d 2e 2f  | !"#$%&'()*+,-./|
[root@fe01 bin]# ./PCIe_DMA_Readout /dev/xdma0_c2h_0 0x00001000 32
Transfer speed: 1.02 MB/s
0000: f7 9f 7f ff ff ff bf ff f7 7e ef fe ff 7f c4 ce  |.........~......|
0010: 2d a3 06 ef 47 1e ec fa a6 7d cc b0 f1 5d 7d 2f  |-...G....}...]}/|
[root@fe01 bin]# ./PCIe_DMA_Readout /dev/xdma0_c2h_0 0x00000400 32
Transfer speed: 1.00 MB/s
0000: 00 00 00 00 01 00 00 00 02 00 00 00 03 00 00 00  |................|
0010: 04 00 00 00 05 00 00 00 06 00 00 00 07 00 00 00  |................|
[root@fe01 bin]# ./PCIe_DMA_Readout /dev/xdma0_c2h_0 0x00000800 32
Transfer speed: 0.21 MB/s
0000: ff ff 00 00 ff fe 00 01 ff fd 00 02 ff fc 00 03  |................|
0010: ff fb 00 04 ff fa 00 05 ff f9 00 06 ff f8 00 07  |................|
[root@fe01 bin]# ./PCIe_DMA_Readout /dev/xdma0_c2h_0 0x00000800 32
[root@fe01 bin]# ./PCIe_DMA_Readout /dev/xdma0_c2h_0 0x00000000 16
Transfer speed: 0.11 MB/s
0000: 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f  |................|
[root@fe01 bin]# ./PCIe_DMA_Readout /dev/xdma0_c2h_0 0x00000000 32
Transfer speed: 1.24 MB/s
0000: 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f  |................|
0010: 20 21 22 23 24 25 26 27 28 29 2a 2b 2c 2d 2e 2f  | !"#$%&'()*+,-./|
[root@fe01 bin]# ./PCIe_DMA_Readout /dev/xdma0_c2h_0 0x00001000 32
Transfer speed: 1.02 MB/s
0000: f7 9f 7f ff ff ff bf ff f7 7e ef fe ff 7f c4 ce  |.........~......|
0010: 2d a3 06 ef 47 1e ec fa a6 7d cc b0 f1 5d 7d 2f  |-...G....}...]}/|
[root@fe01 bin]# ./PCIe_DMA_Readout /dev/xdma0_c2h_0 0x00000400 32
Transfer speed: 1.00 MB/s
0000: 00 00 00 00 01 00 00 00 02 00 00 00 03 00 00 00  |................|
0010: 04 00 00 00 05 00 00 00 06 00 00 00 07 00 00 00  |................|
[root@fe01 bin]# ./PCIe_DMA_Readout /dev/xdma0_c2h_0 0x00000800 32
Transfer speed: 0.21 MB/s
0000: ff ff 00 00 ff fe 00 01 ff fd 00 02 ff fc 00 03  |................|
0010: ff fb 00 04 ff fa 00 05 ff f9 00 06 ff f8 00 07  |................|
[root@fe01 bin]# ./PCIe_DMA_Readout /dev/xdma0_c2h_0 0x00000800 32

12/09/2024 04:56

I updated fe01 to ALMA9 Linux. This updated the kernel. The XIlinx XDMA repository is not up to date for our kenel version given by:

[root@dhcp-10-163-102-46 drivers]# uname -r
5.14.0-427.33.1.el9_4.x86_64
[root@dhcp-10-163-102-46 drivers]# uname -r
5.14.0-427.33.1.el9_4.x86_64

However, there is a pull request that updates the driver for the newer kernel methods. Which can be added with:

git remote add upstream https://github.com/Xilinx/dma_ip_drivers.git
git fetch upstream pull/142/head:pull-142
git checkout pull-142
git remote add upstream https://github.com/Xilinx/dma_ip_drivers.git
git fetch upstream pull/142/head:pull-142
git checkout pull-142

However, this fails:

[root@dhcp-10-163-102-46 xdma]# make install
Makefile:17: XVC_FLAGS: .
make -C /lib/modules/5.14.0-427.33.1.el9_4.x86_64/build M=/home/drivers/dma_ip_drivers/XDMA/linux-kernel/xdma modules
make[1]: Entering directory '/usr/src/kernels/5.14.0-427.33.1.el9_4.x86_64'
/home/drivers/dma_ip_drivers/XDMA/linux-kernel/xdma/Makefile:17: XVC_FLAGS: .
  CC [M]  /home/drivers/dma_ip_drivers/XDMA/linux-kernel/xdma/xdma_cdev.o
  CC [M]  /home/drivers/dma_ip_drivers/XDMA/linux-kernel/xdma/cdev_ctrl.o
  CC [M]  /home/drivers/dma_ip_drivers/XDMA/linux-kernel/xdma/cdev_sgdma.o
/home/drivers/dma_ip_drivers/XDMA/linux-kernel/xdma/cdev_sgdma.c: In function ‘async_io_handler’:
/home/drivers/dma_ip_drivers/XDMA/linux-kernel/xdma/cdev_sgdma.c:111:17: error: too many arguments to function ‘caio->iocb->ki_complete’
  111 |                 caio->iocb->ki_complete(caio->iocb, res, res2);
      |                 ^~~~
/home/drivers/dma_ip_drivers/XDMA/linux-kernel/xdma/cdev_sgdma.c:130:9: error: too many arguments to function ‘caio->iocb->ki_complete’
  130 |         caio->iocb->ki_complete(caio->iocb, numbytes, -EBUSY);
      |         ^~~~
/home/drivers/dma_ip_drivers/XDMA/linux-kernel/xdma/cdev_sgdma.c: In function ‘cdev_write_iter’:
/home/drivers/dma_ip_drivers/XDMA/linux-kernel/xdma/cdev_sgdma.c:574:41: error: ‘struct iov_iter’ has no member named ‘iov’; did you mean ‘__iov’?
  574 |         return cdev_aio_write(iocb, io->iov, io->nr_segs, io->iov_offset);
      |                                         ^~~
      |                                         __iov
/home/drivers/dma_ip_drivers/XDMA/linux-kernel/xdma/cdev_sgdma.c: In function ‘cdev_read_iter’:
/home/drivers/dma_ip_drivers/XDMA/linux-kernel/xdma/cdev_sgdma.c:583:40: error: ‘struct iov_iter’ has no member named ‘iov’; did you mean ‘__iov’?
  583 |         return cdev_aio_read(iocb, io->iov, io->nr_segs, io->iov_offset);
      |                                        ^~~
      |                                        __iov
/home/drivers/dma_ip_drivers/XDMA/linux-kernel/xdma/cdev_sgdma.c: In function ‘cdev_write_iter’:
/home/drivers/dma_ip_drivers/XDMA/linux-kernel/xdma/cdev_sgdma.c:578:1: error: control reaches end of non-void function [-Werror=return-type]
  578 | }
      | ^
/home/drivers/dma_ip_drivers/XDMA/linux-kernel/xdma/cdev_sgdma.c: In function ‘cdev_read_iter’:
/home/drivers/dma_ip_drivers/XDMA/linux-kernel/xdma/cdev_sgdma.c:587:1: error: control reaches end of non-void function [-Werror=return-type]
  587 | }
      | ^
cc1: some warnings being treated as errors
make[2]: *** [scripts/Makefile.build:299: /home/drivers/dma_ip_drivers/XDMA/linux-kernel/xdma/cdev_sgdma.o] Error 1
make[1]: *** [Makefile:1936: /home/drivers/dma_ip_drivers/XDMA/linux-kernel/xdma] Error 2
make[1]: Leaving directory '/usr/src/kernels/5.14.0-427.33.1.el9_4.x86_64'
make: *** [Makefile:39: all] Error 2
[root@dhcp-10-163-102-46 xdma]# make install
Makefile:17: XVC_FLAGS: .
make -C /lib/modules/5.14.0-427.33.1.el9_4.x86_64/build M=/home/drivers/dma_ip_drivers/XDMA/linux-kernel/xdma modules
make[1]: Entering directory '/usr/src/kernels/5.14.0-427.33.1.el9_4.x86_64'
/home/drivers/dma_ip_drivers/XDMA/linux-kernel/xdma/Makefile:17: XVC_FLAGS: .
  CC [M]  /home/drivers/dma_ip_drivers/XDMA/linux-kernel/xdma/xdma_cdev.o
  CC [M]  /home/drivers/dma_ip_drivers/XDMA/linux-kernel/xdma/cdev_ctrl.o
  CC [M]  /home/drivers/dma_ip_drivers/XDMA/linux-kernel/xdma/cdev_sgdma.o
/home/drivers/dma_ip_drivers/XDMA/linux-kernel/xdma/cdev_sgdma.c: In function ‘async_io_handler’:
/home/drivers/dma_ip_drivers/XDMA/linux-kernel/xdma/cdev_sgdma.c:111:17: error: too many arguments to function ‘caio->iocb->ki_complete’
  111 |                 caio->iocb->ki_complete(caio->iocb, res, res2);
      |                 ^~~~
/home/drivers/dma_ip_drivers/XDMA/linux-kernel/xdma/cdev_sgdma.c:130:9: error: too many arguments to function ‘caio->iocb->ki_complete’
  130 |         caio->iocb->ki_complete(caio->iocb, numbytes, -EBUSY);
      |         ^~~~
/home/drivers/dma_ip_drivers/XDMA/linux-kernel/xdma/cdev_sgdma.c: In function ‘cdev_write_iter’:
/home/drivers/dma_ip_drivers/XDMA/linux-kernel/xdma/cdev_sgdma.c:574:41: error: ‘struct iov_iter’ has no member named ‘iov’; did you mean ‘__iov’?
  574 |         return cdev_aio_write(iocb, io->iov, io->nr_segs, io->iov_offset);
      |                                         ^~~
      |                                         __iov
/home/drivers/dma_ip_drivers/XDMA/linux-kernel/xdma/cdev_sgdma.c: In function ‘cdev_read_iter’:
/home/drivers/dma_ip_drivers/XDMA/linux-kernel/xdma/cdev_sgdma.c:583:40: error: ‘struct iov_iter’ has no member named ‘iov’; did you mean ‘__iov’?
  583 |         return cdev_aio_read(iocb, io->iov, io->nr_segs, io->iov_offset);
      |                                        ^~~
      |                                        __iov
/home/drivers/dma_ip_drivers/XDMA/linux-kernel/xdma/cdev_sgdma.c: In function ‘cdev_write_iter’:
/home/drivers/dma_ip_drivers/XDMA/linux-kernel/xdma/cdev_sgdma.c:578:1: error: control reaches end of non-void function [-Werror=return-type]
  578 | }
      | ^
/home/drivers/dma_ip_drivers/XDMA/linux-kernel/xdma/cdev_sgdma.c: In function ‘cdev_read_iter’:
/home/drivers/dma_ip_drivers/XDMA/linux-kernel/xdma/cdev_sgdma.c:587:1: error: control reaches end of non-void function [-Werror=return-type]
  587 | }
      | ^
cc1: some warnings being treated as errors
make[2]: *** [scripts/Makefile.build:299: /home/drivers/dma_ip_drivers/XDMA/linux-kernel/xdma/cdev_sgdma.o] Error 1
make[1]: *** [Makefile:1936: /home/drivers/dma_ip_drivers/XDMA/linux-kernel/xdma] Error 2
make[1]: Leaving directory '/usr/src/kernels/5.14.0-427.33.1.el9_4.x86_64'
make: *** [Makefile:39: all] Error 2

It seems there linux versions cutoffs are wrong? I went into c_dev_sgma.c and manually changed these lines:

c_dev_sgma.c::108:

-#if LINUX_VERSION_CODE >= KERNEL_VERSION(5, 16, 0)
+#if LINUX_VERSION_CODE >= KERNEL_VERSION(5, 13, 0)
-#if LINUX_VERSION_CODE >= KERNEL_VERSION(5, 16, 0)
+#if LINUX_VERSION_CODE >= KERNEL_VERSION(5, 13, 0)

c_dev_sgma.c::127:

-#if LINUX_VERSION_CODE >= KERNEL_VERSION(5, 16, 0)
+#if LINUX_VERSION_CODE >= KERNEL_VERSION(5, 13, 0)
-#if LINUX_VERSION_CODE >= KERNEL_VERSION(5, 16, 0)
+#if LINUX_VERSION_CODE >= KERNEL_VERSION(5, 13, 0)

c_dev_sgma.c::573:

-#if LINUX_VERSION_CODE >= KERNEL_VERSION(6, 4, 0)
+#if LINUX_VERSION_CODE >= KERNEL_VERSION(5, 4, 0)
-#if LINUX_VERSION_CODE >= KERNEL_VERSION(6, 4, 0)
+#if LINUX_VERSION_CODE >= KERNEL_VERSION(5, 4, 0)

c_dev_sgma.c::582:

-#if LINUX_VERSION_CODE >= KERNEL_VERSION(6, 4, 0)
+#if LINUX_VERSION_CODE >= KERNEL_VERSION(5, 4, 0)
-#if LINUX_VERSION_CODE >= KERNEL_VERSION(6, 4, 0)
+#if LINUX_VERSION_CODE >= KERNEL_VERSION(5, 4, 0)

and then I was able to make install:

[root@dhcp-10-163-102-46 xdma]# make install
Makefile:17: XVC_FLAGS: .
make -C /lib/modules/5.14.0-427.33.1.el9_4.x86_64/build M=/home/drivers/dma_ip_drivers/XDMA/linux-kernel/xdma modules
make[1]: Entering directory '/usr/src/kernels/5.14.0-427.33.1.el9_4.x86_64'
/home/drivers/dma_ip_drivers/XDMA/linux-kernel/xdma/Makefile:17: XVC_FLAGS: .
  CC [M]  /home/drivers/dma_ip_drivers/XDMA/linux-kernel/xdma/cdev_sgdma.o
  CC [M]  /home/drivers/dma_ip_drivers/XDMA/linux-kernel/xdma/cdev_xvc.o
  CC [M]  /home/drivers/dma_ip_drivers/XDMA/linux-kernel/xdma/cdev_bypass.o
  CC [M]  /home/drivers/dma_ip_drivers/XDMA/linux-kernel/xdma/xdma_mod.o
  CC [M]  /home/drivers/dma_ip_drivers/XDMA/linux-kernel/xdma/xdma_thread.o
  LD [M]  /home/drivers/dma_ip_drivers/XDMA/linux-kernel/xdma/xdma.o
/home/drivers/dma_ip_drivers/XDMA/linux-kernel/xdma/Makefile:17: XVC_FLAGS: .
  MODPOST /home/drivers/dma_ip_drivers/XDMA/linux-kernel/xdma/Module.symvers
  CC [M]  /home/drivers/dma_ip_drivers/XDMA/linux-kernel/xdma/xdma.mod.o
  LD [M]  /home/drivers/dma_ip_drivers/XDMA/linux-kernel/xdma/xdma.ko
  BTF [M] /home/drivers/dma_ip_drivers/XDMA/linux-kernel/xdma/xdma.ko
Skipping BTF generation for /home/drivers/dma_ip_drivers/XDMA/linux-kernel/xdma/xdma.ko due to unavailability of vmlinux
make[1]: Leaving directory '/usr/src/kernels/5.14.0-427.33.1.el9_4.x86_64'
make -C /lib/modules/5.14.0-427.33.1.el9_4.x86_64/build M=/home/drivers/dma_ip_drivers/XDMA/linux-kernel/xdma modules_install
make[1]: Entering directory '/usr/src/kernels/5.14.0-427.33.1.el9_4.x86_64'
  INSTALL /lib/modules/5.14.0-427.33.1.el9_4.x86_64/extra/xdma.ko
  SIGN    /lib/modules/5.14.0-427.33.1.el9_4.x86_64/extra/xdma.ko
At main.c:167:
- SSL error:FFFFFFFF80000002:system library::No such file or directory: crypto/bio/bss_file.c:67
- SSL error:10000080:BIO routines::no such file: crypto/bio/bss_file.c:75
sign-file: certs/signing_key.pem: No such file or directory
  DEPMOD  /lib/modules/5.14.0-427.33.1.el9_4.x86_64
make[1]: Leaving directory '/usr/src/kernels/5.14.0-427.33.1.el9_4.x86_64'
[root@dhcp-10-163-102-46 xdma]#
[root@dhcp-10-163-102-46 xdma]# make install
Makefile:17: XVC_FLAGS: .
make -C /lib/modules/5.14.0-427.33.1.el9_4.x86_64/build M=/home/drivers/dma_ip_drivers/XDMA/linux-kernel/xdma modules
make[1]: Entering directory '/usr/src/kernels/5.14.0-427.33.1.el9_4.x86_64'
/home/drivers/dma_ip_drivers/XDMA/linux-kernel/xdma/Makefile:17: XVC_FLAGS: .
  CC [M]  /home/drivers/dma_ip_drivers/XDMA/linux-kernel/xdma/cdev_sgdma.o
  CC [M]  /home/drivers/dma_ip_drivers/XDMA/linux-kernel/xdma/cdev_xvc.o
  CC [M]  /home/drivers/dma_ip_drivers/XDMA/linux-kernel/xdma/cdev_bypass.o
  CC [M]  /home/drivers/dma_ip_drivers/XDMA/linux-kernel/xdma/xdma_mod.o
  CC [M]  /home/drivers/dma_ip_drivers/XDMA/linux-kernel/xdma/xdma_thread.o
  LD [M]  /home/drivers/dma_ip_drivers/XDMA/linux-kernel/xdma/xdma.o
/home/drivers/dma_ip_drivers/XDMA/linux-kernel/xdma/Makefile:17: XVC_FLAGS: .
  MODPOST /home/drivers/dma_ip_drivers/XDMA/linux-kernel/xdma/Module.symvers
  CC [M]  /home/drivers/dma_ip_drivers/XDMA/linux-kernel/xdma/xdma.mod.o
  LD [M]  /home/drivers/dma_ip_drivers/XDMA/linux-kernel/xdma/xdma.ko
  BTF [M] /home/drivers/dma_ip_drivers/XDMA/linux-kernel/xdma/xdma.ko
Skipping BTF generation for /home/drivers/dma_ip_drivers/XDMA/linux-kernel/xdma/xdma.ko due to unavailability of vmlinux
make[1]: Leaving directory '/usr/src/kernels/5.14.0-427.33.1.el9_4.x86_64'
make -C /lib/modules/5.14.0-427.33.1.el9_4.x86_64/build M=/home/drivers/dma_ip_drivers/XDMA/linux-kernel/xdma modules_install
make[1]: Entering directory '/usr/src/kernels/5.14.0-427.33.1.el9_4.x86_64'
  INSTALL /lib/modules/5.14.0-427.33.1.el9_4.x86_64/extra/xdma.ko
  SIGN    /lib/modules/5.14.0-427.33.1.el9_4.x86_64/extra/xdma.ko
At main.c:167:
- SSL error:FFFFFFFF80000002:system library::No such file or directory: crypto/bio/bss_file.c:67
- SSL error:10000080:BIO routines::no such file: crypto/bio/bss_file.c:75
sign-file: certs/signing_key.pem: No such file or directory
  DEPMOD  /lib/modules/5.14.0-427.33.1.el9_4.x86_64
make[1]: Leaving directory '/usr/src/kernels/5.14.0-427.33.1.el9_4.x86_64'
[root@dhcp-10-163-102-46 xdma]#
[root@dhcp-10-163-102-46 linux-kernel]# cd tests/
[root@dhcp-10-163-102-46 tests]# ls
data  dma_memory_mapped_test.sh  dma_streaming_test.sh  load_driver.sh  perform_hwcount.sh  run_test.sh  scripts_mm
[root@dhcp-10-163-102-46 tests]# ./load_driver.sh
interrupt_selection .
Loading driver...insmod xdma.ko interrupt_mode=2 ...

The Kernel module installed correctly and the xmda devices were recognized.
DONE
[root@dhcp-10-163-102-46 tests]# ./run_test.sh
Info: Number of enabled h2c channels = 2
Info: Number of enabled c2h channels = 2
Info: The PCIe DMA core is memory mapped.
Info: Running PCIe DMA memory mapped write read test
        transfer size:  1024, count: 1
Info: Writing to h2c channel 0 at address offset 0.
Info: Writing to h2c channel 1 at address offset 1024.
Info: Wait for current transactions to complete.
/dev/xdma0_h2c_1 ** Average BW = 1024, 2.040885
/dev/xdma0_h2c_0 ** Average BW = 1024, 1.269210
Info: Writing to h2c channel 0 at address offset 2048.
Info: Writing to h2c channel 1 at address offset 3072.
Info: Wait for current transactions to complete.
/dev/xdma0_h2c_0 ** Average BW = 1024, 1.371362
/dev/xdma0_h2c_1 ** Average BW = 1024, 1.563163
Info: Reading from c2h channel 0 at  address offset 0.
Info: Reading from c2h channel 1 at  address offset 1024.
Info: Wait for current transactions to complete.
/dev/xdma0_c2h_0 ** Average BW = 1024, 0.610432
/dev/xdma0_c2h_1 ** Average BW = 1024, 9.607986
Info: Reading from c2h channel 0 at  address offset 2048.
Info: Reading from c2h channel 1 at  address offset 3072.
Info: Wait for current transactions to complete.
/dev/xdma0_c2h_0 ** Average BW = 1024, 3.289019
/dev/xdma0_c2h_1 ** Average BW = 1024, 3.509698
Info: Checking data integrity.
Info: Data check passed for address range  0 - 1024
Info: Data check passed for address range  1024 - 2048
Info: Data check passed for address range  2048 - 3072
Info: Data check passed for address range  3072 - 4096
Info: All PCIe DMA memory mapped tests passed.
Info: All tests in run_tests.sh passed.
[root@dhcp-10-163-102-46 linux-kernel]# cd tests/
[root@dhcp-10-163-102-46 tests]# ls
data  dma_memory_mapped_test.sh  dma_streaming_test.sh  load_driver.sh  perform_hwcount.sh  run_test.sh  scripts_mm
[root@dhcp-10-163-102-46 tests]# ./load_driver.sh
interrupt_selection .
Loading driver...insmod xdma.ko interrupt_mode=2 ...

The Kernel module installed correctly and the xmda devices were recognized.
DONE
[root@dhcp-10-163-102-46 tests]# ./run_test.sh
Info: Number of enabled h2c channels = 2
Info: Number of enabled c2h channels = 2
Info: The PCIe DMA core is memory mapped.
Info: Running PCIe DMA memory mapped write read test
        transfer size:  1024, count: 1
Info: Writing to h2c channel 0 at address offset 0.
Info: Writing to h2c channel 1 at address offset 1024.
Info: Wait for current transactions to complete.
/dev/xdma0_h2c_1 ** Average BW = 1024, 2.040885
/dev/xdma0_h2c_0 ** Average BW = 1024, 1.269210
Info: Writing to h2c channel 0 at address offset 2048.
Info: Writing to h2c channel 1 at address offset 3072.
Info: Wait for current transactions to complete.
/dev/xdma0_h2c_0 ** Average BW = 1024, 1.371362
/dev/xdma0_h2c_1 ** Average BW = 1024, 1.563163
Info: Reading from c2h channel 0 at  address offset 0.
Info: Reading from c2h channel 1 at  address offset 1024.
Info: Wait for current transactions to complete.
/dev/xdma0_c2h_0 ** Average BW = 1024, 0.610432
/dev/xdma0_c2h_1 ** Average BW = 1024, 9.607986
Info: Reading from c2h channel 0 at  address offset 2048.
Info: Reading from c2h channel 1 at  address offset 3072.
Info: Wait for current transactions to complete.
/dev/xdma0_c2h_0 ** Average BW = 1024, 3.289019
/dev/xdma0_c2h_1 ** Average BW = 1024, 3.509698
Info: Checking data integrity.
Info: Data check passed for address range  0 - 1024
Info: Data check passed for address range  1024 - 2048
Info: Data check passed for address range  2048 - 3072
Info: Data check passed for address range  3072 - 4096
Info: All PCIe DMA memory mapped tests passed.
Info: All tests in run_tests.sh passed.

Thing seem to work in the tests. Also they seem to work in my XDMA library:

[root@dhcp-10-163-102-46 bin]# ./PCIe_DMA_Readout /dev/xdma0_c2h_0 /dev/xdma0_h2c_0 /dev/xdma0_control 0x00000000 32
Write operation successful. Time taken: 0.000468421 seconds.
Transfer speed: 0 MB/s
Read operation successful, 32 bytes read. Time taken: 0.000114502 seconds.
0: ffffffac ffffffac ffffffac ffffffac ffffffac ffffffac ffffffac ffffffac ffffffac ffffffac ffffffac ffffffac ffffffac ffffffac ffffffac ffffffac  |................|
10: ffffffac ffffffac ffffffac ffffffac ffffffac ffffffac ffffffac ffffffac ffffffac ffffffac ffffffac ffffffac ffffffac ffffffac ffffffac ffffffac  |................|
Transfer speed: 0.272418 MB/s
[root@dhcp-10-163-102-46 bin]#
[root@dhcp-10-163-102-46 bin]# ./PCIe_DMA_Readout /dev/xdma0_c2h_0 /dev/xdma0_h2c_0 /dev/xdma0_control 0x00000000 32
Write operation successful. Time taken: 0.000468421 seconds.
Transfer speed: 0 MB/s
Read operation successful, 32 bytes read. Time taken: 0.000114502 seconds.
0: ffffffac ffffffac ffffffac ffffffac ffffffac ffffffac ffffffac ffffffac ffffffac ffffffac ffffffac ffffffac ffffffac ffffffac ffffffac ffffffac  |................|
10: ffffffac ffffffac ffffffac ffffffac ffffffac ffffffac ffffffac ffffffac ffffffac ffffffac ffffffac ffffffac ffffffac ffffffac ffffffac ffffffac  |................|
Transfer speed: 0.272418 MB/s
[root@dhcp-10-163-102-46 bin]#

12/09/2024 06:36

I wrote a frontend that just writes and reads data, incrimenting things a little bit as it goes. It seems to work:

#include <stdio.h>
#include <stdlib.h>
#include <math.h>
#include <string.h>
#include <iostream>
#include <fstream>
#include <sstream>
#include <vector>
#include <chrono>
#include "midas.h"
#include "mfe.h"
#include "xdma_device_read.h"
#include "xdma_device_write.h"

// Define your PCIe devices
XDMADeviceRead deviceRead("/dev/xdma0_c2h_0");
XDMADeviceWrite deviceWrite("/dev/xdma0_h2c_0");

// Globals
const char *frontend_name = "DataSimulator";
const char *frontend_file_name = __FILE__;
BOOL frontend_call_loop = FALSE;
INT display_period = 1000;
INT max_event_size = 1024 * 1024;
INT max_event_size_frag = 5 * max_event_size;
INT event_buffer_size = 5 * max_event_size;

// Define a vector to store 16-bit words
std::vector<int16_t> data;

// Global variable to keep track of the last poll time
std::chrono::steady_clock::time_point last_poll_time;
const std::chrono::microseconds polling_interval(1000*1000);

// Global variable to cycle the write pattern
uint8_t write_value = 0x00; // Starting value

// Function declarations
INT frontend_init(void);
INT frontend_exit(void);
INT begin_of_run(INT run_number, char *error);
INT end_of_run(INT run_number, char *error);
INT pause_run(INT run_number, char *error);
INT resume_run(INT run_number, char *error);
INT frontend_loop(void);

INT read_trigger_event(char *pevent, INT off);
INT read_periodic_event(char *pevent, INT off);

INT poll_event(INT source, INT count, BOOL test);
INT interrupt_configure(INT cmd, INT source, POINTER_T adr);

// Equipment list
BOOL equipment_common_overwrite = TRUE;

EQUIPMENT equipment[] = {
    {"Data Simulator",
        {2, 0,
            "SYSTEM",
            EQ_POLLED,
            0,
            "MIDAS",
            TRUE,
            RO_RUNNING | RO_TRANSITIONS |
            RO_ODB,
            10,
            0,
            0,
            TRUE,
            "", "", "",},
        read_trigger_event
    },

    {""}
};

// Trigger Update
void trigger_update(INT hDB, INT hkey, void*)
{
}

// Frontend Init
int frontend_init() {
    // Initialize PCIe devices
    deviceRead = XDMADeviceRead("/dev/xdma0_c2h_0");
    deviceWrite = XDMADeviceWrite("/dev/xdma0_h2c_0");
    deviceRead.initialize();
    deviceWrite.initialize();

    return SUCCESS;
}

// Frontend Exit
INT frontend_exit()
{
    return SUCCESS;
}

// Begin of Run
INT begin_of_run(INT run_number, char *error)
{
    return SUCCESS;
}

// End of Run
INT end_of_run(INT run_number, char *error)
{
    return SUCCESS;
}

// Pause Run
INT pause_run(INT run_number, char *error)
{
    return SUCCESS;
}

// Resume Run
INT resume_run(INT run_number, char *error)
{
    return SUCCESS;
}

// Frontend Loop
INT frontend_loop()
{
    return SUCCESS;
}

// Poll Event
INT poll_event(INT source, INT count, BOOL test) {
    auto now = std::chrono::steady_clock::now();
    if (now - last_poll_time >= polling_interval) {
        last_poll_time = now;
        return TRUE;
    }
    if (test) {
        return FALSE;
    }
    return FALSE;
}

// Interrupt Configuration
INT interrupt_configure(INT cmd, INT source, POINTER_T adr)
{
    switch (cmd) {
    case CMD_INTERRUPT_ENABLE:
        break;
    case CMD_INTERRUPT_DISABLE:
        break;
    case CMD_INTERRUPT_ATTACH:
        break;
    case CMD_INTERRUPT_DETACH:
        break;
    }
    return SUCCESS;
}

// Event Readout
INT read_trigger_event(char *pevent, INT off)
{
    // Init bank structure
    bk_init32(pevent);

    // Create a bank named "CR00" and specify the data type as TID_SHORT
    short *pdata;
    bk_create(pevent, "CR00", TID_SHORT, (void **)&pdata);

    // Write a cycling pattern to the PCIe device
    size_t size = 1024; // Adjust size as needed
    std::vector<char> buffer(size);
    for (size_t i = 0; i < size; ++i) {
        buffer[i] = write_value;
    }

    // Assuming address is 0 for the write operation, you can adjust as needed
    deviceWrite.writeToDevice(0, size, buffer.data());

    // Increment the write value and wrap around if needed
    write_value += 0x01;
    if (write_value > 0xAA) {
        write_value = 0x00;
    }

    // Read data from the PCIe device
    std::vector<char> read_buffer = deviceRead.readFromDevice(0, size);

    // Convert char buffer to short and copy to pdata
    for (size_t i = 0; i < size / sizeof(short); ++i) {
        // Ensure the buffer has enough data
        if (i * sizeof(short) + sizeof(short) <= read_buffer.size()) {
            *pdata++ = *reinterpret_cast<short*>(&read_buffer[i * sizeof(short)]);
        }
    }

    // Close the bank
    bk_close(pevent, pdata);

    return bk_size(pevent);
}

// Periodic Event
INT read_periodic_event(char *pevent, INT off)
{
    // Init bank structure
    bk_init32(pevent);

    // Create a bank named "CR00" and specify the data type as TID_SHORT
    short *pdata;
    bk_create(pevent, "CR00", TID_SHORT, (void **)&pdata);

    // Read data from the PCIe device
    size_t size = 1024; // Adjust size as needed
    std::vector<char> buffer = deviceRead.readFromDevice(0, size);

    // Convert char buffer to short and copy to pdata
    for (size_t i = 0; i < size / sizeof(short); ++i) {
        // Ensure the buffer has enough data
        if (i * sizeof(short) + sizeof(short) <= buffer.size()) {
            *pdata++ = *reinterpret_cast<short*>(&buffer[i * sizeof(short)]);
        }
    }

    // Close the bank
    bk_close(pevent, pdata);

    return bk_size(pevent);
}

#include <stdio.h>
#include <stdlib.h>
#include <math.h>
#include <string.h>
#include <iostream>
#include <fstream>
#include <sstream>
#include <vector>
#include <chrono>
#include "midas.h"
#include "mfe.h"
#include "xdma_device_read.h"
#include "xdma_device_write.h"

// Define your PCIe devices
XDMADeviceRead deviceRead("/dev/xdma0_c2h_0");
XDMADeviceWrite deviceWrite("/dev/xdma0_h2c_0");

// Globals
const char *frontend_name = "DataSimulator";
const char *frontend_file_name = __FILE__;
BOOL frontend_call_loop = FALSE;
INT display_period = 1000;
INT max_event_size = 1024 * 1024;
INT max_event_size_frag = 5 * max_event_size;
INT event_buffer_size = 5 * max_event_size;

// Define a vector to store 16-bit words
std::vector<int16_t> data;

// Global variable to keep track of the last poll time
std::chrono::steady_clock::time_point last_poll_time;
const std::chrono::microseconds polling_interval(1000*1000);

// Global variable to cycle the write pattern
uint8_t write_value = 0x00; // Starting value

// Function declarations
INT frontend_init(void);
INT frontend_exit(void);
INT begin_of_run(INT run_number, char *error);
INT end_of_run(INT run_number, char *error);
INT pause_run(INT run_number, char *error);
INT resume_run(INT run_number, char *error);
INT frontend_loop(void);

INT read_trigger_event(char *pevent, INT off);
INT read_periodic_event(char *pevent, INT off);

INT poll_event(INT source, INT count, BOOL test);
INT interrupt_configure(INT cmd, INT source, POINTER_T adr);

// Equipment list
BOOL equipment_common_overwrite = TRUE;

EQUIPMENT equipment[] = {
    {"Data Simulator",
        {2, 0,
            "SYSTEM",
            EQ_POLLED,
            0,
            "MIDAS",
            TRUE,
            RO_RUNNING | RO_TRANSITIONS |
            RO_ODB,
            10,
            0,
            0,
            TRUE,
            "", "", "",},
        read_trigger_event
    },

    {""}
};

// Trigger Update
void trigger_update(INT hDB, INT hkey, void*)
{
}

// Frontend Init
int frontend_init() {
    // Initialize PCIe devices
    deviceRead = XDMADeviceRead("/dev/xdma0_c2h_0");
    deviceWrite = XDMADeviceWrite("/dev/xdma0_h2c_0");
    deviceRead.initialize();
    deviceWrite.initialize();

    return SUCCESS;
}

// Frontend Exit
INT frontend_exit()
{
    return SUCCESS;
}

// Begin of Run
INT begin_of_run(INT run_number, char *error)
{
    return SUCCESS;
}

// End of Run
INT end_of_run(INT run_number, char *error)
{
    return SUCCESS;
}

// Pause Run
INT pause_run(INT run_number, char *error)
{
    return SUCCESS;
}

// Resume Run
INT resume_run(INT run_number, char *error)
{
    return SUCCESS;
}

// Frontend Loop
INT frontend_loop()
{
    return SUCCESS;
}

// Poll Event
INT poll_event(INT source, INT count, BOOL test) {
    auto now = std::chrono::steady_clock::now();
    if (now - last_poll_time >= polling_interval) {
        last_poll_time = now;
        return TRUE;
    }
    if (test) {
        return FALSE;
    }
    return FALSE;
}

// Interrupt Configuration
INT interrupt_configure(INT cmd, INT source, POINTER_T adr)
{
    switch (cmd) {
    case CMD_INTERRUPT_ENABLE:
        break;
    case CMD_INTERRUPT_DISABLE:
        break;
    case CMD_INTERRUPT_ATTACH:
        break;
    case CMD_INTERRUPT_DETACH:
        break;
    }
    return SUCCESS;
}

// Event Readout
INT read_trigger_event(char *pevent, INT off)
{
    // Init bank structure
    bk_init32(pevent);

    // Create a bank named "CR00" and specify the data type as TID_SHORT
    short *pdata;
    bk_create(pevent, "CR00", TID_SHORT, (void **)&pdata);

    // Write a cycling pattern to the PCIe device
    size_t size = 1024; // Adjust size as needed
    std::vector<char> buffer(size);
    for (size_t i = 0; i < size; ++i) {
        buffer[i] = write_value;
    }

    // Assuming address is 0 for the write operation, you can adjust as needed
    deviceWrite.writeToDevice(0, size, buffer.data());

    // Increment the write value and wrap around if needed
    write_value += 0x01;
    if (write_value > 0xAA) {
        write_value = 0x00;
    }

    // Read data from the PCIe device
    std::vector<char> read_buffer = deviceRead.readFromDevice(0, size);

    // Convert char buffer to short and copy to pdata
    for (size_t i = 0; i < size / sizeof(short); ++i) {
        // Ensure the buffer has enough data
        if (i * sizeof(short) + sizeof(short) <= read_buffer.size()) {
            *pdata++ = *reinterpret_cast<short*>(&read_buffer[i * sizeof(short)]);
        }
    }

    // Close the bank
    bk_close(pevent, pdata);

    return bk_size(pevent);
}

// Periodic Event
INT read_periodic_event(char *pevent, INT off)
{
    // Init bank structure
    bk_init32(pevent);

    // Create a bank named "CR00" and specify the data type as TID_SHORT
    short *pdata;
    bk_create(pevent, "CR00", TID_SHORT, (void **)&pdata);

    // Read data from the PCIe device
    size_t size = 1024; // Adjust size as needed
    std::vector<char> buffer = deviceRead.readFromDevice(0, size);

    // Convert char buffer to short and copy to pdata
    for (size_t i = 0; i < size / sizeof(short); ++i) {
        // Ensure the buffer has enough data
        if (i * sizeof(short) + sizeof(short) <= buffer.size()) {
            *pdata++ = *reinterpret_cast<short*>(&buffer[i * sizeof(short)]);
        }
    }

    // Close the bank
    bk_close(pevent, pdata);

    return bk_size(pevent);
}

With mdump outputs:

[root@dhcp-10-163-102-46 pcie_data_simulator]# $MIDASSYS/bin/mdump
- MIDAS revision: Fri Feb 2 18:49:36 2024 -0500 -  on branch pioneerMidas -- Enter <!> to Exit ------- Midas Dump ---
------------------------ Event# 1 ------------------------
Evid:0002- Mask:0000- Serial:54- Time:0x66e2c434- Dsize:1044/0x414
#banks:1 - Bank list:-CR00-

Bank:CR00 Length: 1024(I*1)/256(I*4)/512(Type) Type:Signed Integer*2
   1-> 13878 13878 13878 13878 13878 13878 13878 13878
   9-> 13878 13878 13878 13878 13878 13878 13878 13878
  17-> 13878 13878 13878 13878 13878 13878 13878 13878
  25-> 13878 13878 13878 13878 13878 13878 13878 13878
  33-> 13878 13878 13878 13878 13878 13878 13878 13878
  41-> 13878 13878 13878 13878 13878 13878 13878 13878
  49-> 13878 13878 13878 13878 13878 13878 13878 13878
  57-> 13878 13878 13878 13878 13878 13878 13878 13878
  65-> 13878 13878 13878 13878 13878 13878 13878 13878
  73-> 13878 13878 13878 13878 13878 13878 13878 13878
  81-> 13878 13878 13878 13878 13878 13878 13878 13878
  89-> 13878 13878 13878 13878 13878 13878 13878 13878
  97-> 13878 13878 13878 13878 13878 13878 13878 13878
 105-> 13878 13878 13878 13878 13878 13878 13878 13878
 113-> 13878 13878 13878 13878 13878 13878 13878 13878
 121-> 13878 13878 13878 13878 13878 13878 13878 13878
 129-> 13878 13878 13878 13878 13878 13878 13878 13878
 137-> 13878 13878 13878 13878 13878 13878 13878 13878
 145-> 13878 13878 13878 13878 13878 13878 13878 13878
 153-> 13878 13878 13878 13878 13878 13878 13878 13878
 161-> 13878 13878 13878 13878 13878 13878 13878 13878
 169-> 13878 13878 13878 13878 13878 13878 13878 13878
 177-> 13878 13878 13878 13878 13878 13878 13878 13878
 185-> 13878 13878 13878 13878 13878 13878 13878 13878
 193-> 13878 13878 13878 13878 13878 13878 13878 13878
 201-> 13878 13878 13878 13878 13878 13878 13878 13878
 209-> 13878 13878 13878 13878 13878 13878 13878 13878
 217-> 13878 13878 13878 13878 13878 13878 13878 13878
 225-> 13878 13878 13878 13878 13878 13878 13878 13878
 233-> 13878 13878 13878 13878 13878 13878 13878 13878
 241-> 13878 13878 13878 13878 13878 13878 13878 13878
 249-> 13878 13878 13878 13878 13878 13878 13878 13878
 257-> 13878 13878 13878 13878 13878 13878 13878 13878
 265-> 13878 13878 13878 13878 13878 13878 13878 13878
 273-> 13878 13878 13878 13878 13878 13878 13878 13878
 281-> 13878 13878 13878 13878 13878 13878 13878 13878
 289-> 13878 13878 13878 13878 13878 13878 13878 13878
 297-> 13878 13878 13878 13878 13878 13878 13878 13878
 305-> 13878 13878 13878 13878 13878 13878 13878 13878
 313-> 13878 13878 13878 13878 13878 13878 13878 13878
 321-> 13878 13878 13878 13878 13878 13878 13878 13878
 329-> 13878 13878 13878 13878 13878 13878 13878 13878
 337-> 13878 13878 13878 13878 13878 13878 13878 13878
 345-> 13878 13878 13878 13878 13878 13878 13878 13878
 353-> 13878 13878 13878 13878 13878 13878 13878 13878
 361-> 13878 13878 13878 13878 13878 13878 13878 13878
 369-> 13878 13878 13878 13878 13878 13878 13878 13878
 377-> 13878 13878 13878 13878 13878 13878 13878 13878
 385-> 13878 13878 13878 13878 13878 13878 13878 13878
 393-> 13878 13878 13878 13878 13878 13878 13878 13878
 401-> 13878 13878 13878 13878 13878 13878 13878 13878
 409-> 13878 13878 13878 13878 13878 13878 13878 13878
 417-> 13878 13878 13878 13878 13878 13878 13878 13878
 425-> 13878 13878 13878 13878 13878 13878 13878 13878
 433-> 13878 13878 13878 13878 13878 13878 13878 13878
 441-> 13878 13878 13878 13878 13878 13878 13878 13878
 449-> 13878 13878 13878 13878 13878 13878 13878 13878
 457-> 13878 13878 13878 13878 13878 13878 13878 13878
 465-> 13878 13878 13878 13878 13878 13878 13878 13878
 473-> 13878 13878 13878 13878 13878 13878 13878 13878
 481-> 13878 13878 13878 13878 13878 13878 13878 13878
 489-> 13878 13878 13878 13878 13878 13878 13878 13878
 497-> 13878 13878 13878 13878 13878 13878 13878 13878
 505-> 13878 13878 13878 13878 13878 13878 13878 13878
[root@dhcp-10-163-102-46 pcie_data_simulator]# $MIDASSYS/bin/mdump
- MIDAS revision: Fri Feb 2 18:49:36 2024 -0500 -  on branch pioneerMidas -- Enter <!> to Exit ------- Midas Dump ---
------------------------ Event# 1 ------------------------
Evid:0002- Mask:0000- Serial:55- Time:0x66e2c435- Dsize:1044/0x414
#banks:1 - Bank list:-CR00-

Bank:CR00 Length: 1024(I*1)/256(I*4)/512(Type) Type:Signed Integer*2
   1-> 14135 14135 14135 14135 14135 14135 14135 14135
   9-> 14135 14135 14135 14135 14135 14135 14135 14135
  17-> 14135 14135 14135 14135 14135 14135 14135 14135
  25-> 14135 14135 14135 14135 14135 14135 14135 14135
  33-> 14135 14135 14135 14135 14135 14135 14135 14135
  41-> 14135 14135 14135 14135 14135 14135 14135 14135
  49-> 14135 14135 14135 14135 14135 14135 14135 14135
  57-> 14135 14135 14135 14135 14135 14135 14135 14135
  65-> 14135 14135 14135 14135 14135 14135 14135 14135
  73-> 14135 14135 14135 14135 14135 14135 14135 14135
  81-> 14135 14135 14135 14135 14135 14135 14135 14135
  89-> 14135 14135 14135 14135 14135 14135 14135 14135
  97-> 14135 14135 14135 14135 14135 14135 14135 14135
 105-> 14135 14135 14135 14135 14135 14135 14135 14135
 113-> 14135 14135 14135 14135 14135 14135 14135 14135
 121-> 14135 14135 14135 14135 14135 14135 14135 14135
 129-> 14135 14135 14135 14135 14135 14135 14135 14135
 137-> 14135 14135 14135 14135 14135 14135 14135 14135
 145-> 14135 14135 14135 14135 14135 14135 14135 14135
 153-> 14135 14135 14135 14135 14135 14135 14135 14135
 161-> 14135 14135 14135 14135 14135 14135 14135 14135
 169-> 14135 14135 14135 14135 14135 14135 14135 14135
 177-> 14135 14135 14135 14135 14135 14135 14135 14135
 185-> 14135 14135 14135 14135 14135 14135 14135 14135
 193-> 14135 14135 14135 14135 14135 14135 14135 14135
 201-> 14135 14135 14135 14135 14135 14135 14135 14135
 209-> 14135 14135 14135 14135 14135 14135 14135 14135
 217-> 14135 14135 14135 14135 14135 14135 14135 14135
 225-> 14135 14135 14135 14135 14135 14135 14135 14135
 233-> 14135 14135 14135 14135 14135 14135 14135 14135
 241-> 14135 14135 14135 14135 14135 14135 14135 14135
 249-> 14135 14135 14135 14135 14135 14135 14135 14135
 257-> 14135 14135 14135 14135 14135 14135 14135 14135
 265-> 14135 14135 14135 14135 14135 14135 14135 14135
 273-> 14135 14135 14135 14135 14135 14135 14135 14135
 281-> 14135 14135 14135 14135 14135 14135 14135 14135
 289-> 14135 14135 14135 14135 14135 14135 14135 14135
 297-> 14135 14135 14135 14135 14135 14135 14135 14135
 305-> 14135 14135 14135 14135 14135 14135 14135 14135
 313-> 14135 14135 14135 14135 14135 14135 14135 14135
 321-> 14135 14135 14135 14135 14135 14135 14135 14135
 329-> 14135 14135 14135 14135 14135 14135 14135 14135
 337-> 14135 14135 14135 14135 14135 14135 14135 14135
 345-> 14135 14135 14135 14135 14135 14135 14135 14135
 353-> 14135 14135 14135 14135 14135 14135 14135 14135
 361-> 14135 14135 14135 14135 14135 14135 14135 14135
 369-> 14135 14135 14135 14135 14135 14135 14135 14135
 377-> 14135 14135 14135 14135 14135 14135 14135 14135
 385-> 14135 14135 14135 14135 14135 14135 14135 14135
 393-> 14135 14135 14135 14135 14135 14135 14135 14135
 401-> 14135 14135 14135 14135 14135 14135 14135 14135
 409-> 14135 14135 14135 14135 14135 14135 14135 14135
 417-> 14135 14135 14135 14135 14135 14135 14135 14135
 425-> 14135 14135 14135 14135 14135 14135 14135 14135
 433-> 14135 14135 14135 14135 14135 14135 14135 14135
 441-> 14135 14135 14135 14135 14135 14135 14135 14135
 449-> 14135 14135 14135 14135 14135 14135 14135 14135
 457-> 14135 14135 14135 14135 14135 14135 14135 14135
 465-> 14135 14135 14135 14135 14135 14135 14135 14135
 473-> 14135 14135 14135 14135 14135 14135 14135 14135
 481-> 14135 14135 14135 14135 14135 14135 14135 14135
 489-> 14135 14135 14135 14135 14135 14135 14135 14135
 497-> 14135 14135 14135 14135 14135 14135 14135 14135
 505-> 14135 14135 14135 14135 14135 14135 14135 14135
[root@dhcp-10-163-102-46 pcie_data_simulator]#
[root@dhcp-10-163-102-46 pcie_data_simulator]# $MIDASSYS/bin/mdump
- MIDAS revision: Fri Feb 2 18:49:36 2024 -0500 -  on branch pioneerMidas -- Enter <!> to Exit ------- Midas Dump ---
------------------------ Event# 1 ------------------------
Evid:0002- Mask:0000- Serial:54- Time:0x66e2c434- Dsize:1044/0x414
#banks:1 - Bank list:-CR00-

Bank:CR00 Length: 1024(I*1)/256(I*4)/512(Type) Type:Signed Integer*2
   1-> 13878 13878 13878 13878 13878 13878 13878 13878
   9-> 13878 13878 13878 13878 13878 13878 13878 13878
  17-> 13878 13878 13878 13878 13878 13878 13878 13878
  25-> 13878 13878 13878 13878 13878 13878 13878 13878
  33-> 13878 13878 13878 13878 13878 13878 13878 13878
  41-> 13878 13878 13878 13878 13878 13878 13878 13878
  49-> 13878 13878 13878 13878 13878 13878 13878 13878
  57-> 13878 13878 13878 13878 13878 13878 13878 13878
  65-> 13878 13878 13878 13878 13878 13878 13878 13878
  73-> 13878 13878 13878 13878 13878 13878 13878 13878
  81-> 13878 13878 13878 13878 13878 13878 13878 13878
  89-> 13878 13878 13878 13878 13878 13878 13878 13878
  97-> 13878 13878 13878 13878 13878 13878 13878 13878
 105-> 13878 13878 13878 13878 13878 13878 13878 13878
 113-> 13878 13878 13878 13878 13878 13878 13878 13878
 121-> 13878 13878 13878 13878 13878 13878 13878 13878
 129-> 13878 13878 13878 13878 13878 13878 13878 13878
 137-> 13878 13878 13878 13878 13878 13878 13878 13878
 145-> 13878 13878 13878 13878 13878 13878 13878 13878
 153-> 13878 13878 13878 13878 13878 13878 13878 13878
 161-> 13878 13878 13878 13878 13878 13878 13878 13878
 169-> 13878 13878 13878 13878 13878 13878 13878 13878
 177-> 13878 13878 13878 13878 13878 13878 13878 13878
 185-> 13878 13878 13878 13878 13878 13878 13878 13878
 193-> 13878 13878 13878 13878 13878 13878 13878 13878
 201-> 13878 13878 13878 13878 13878 13878 13878 13878
 209-> 13878 13878 13878 13878 13878 13878 13878 13878
 217-> 13878 13878 13878 13878 13878 13878 13878 13878
 225-> 13878 13878 13878 13878 13878 13878 13878 13878
 233-> 13878 13878 13878 13878 13878 13878 13878 13878
 241-> 13878 13878 13878 13878 13878 13878 13878 13878
 249-> 13878 13878 13878 13878 13878 13878 13878 13878
 257-> 13878 13878 13878 13878 13878 13878 13878 13878
 265-> 13878 13878 13878 13878 13878 13878 13878 13878
 273-> 13878 13878 13878 13878 13878 13878 13878 13878
 281-> 13878 13878 13878 13878 13878 13878 13878 13878
 289-> 13878 13878 13878 13878 13878 13878 13878 13878
 297-> 13878 13878 13878 13878 13878 13878 13878 13878
 305-> 13878 13878 13878 13878 13878 13878 13878 13878
 313-> 13878 13878 13878 13878 13878 13878 13878 13878
 321-> 13878 13878 13878 13878 13878 13878 13878 13878
 329-> 13878 13878 13878 13878 13878 13878 13878 13878
 337-> 13878 13878 13878 13878 13878 13878 13878 13878
 345-> 13878 13878 13878 13878 13878 13878 13878 13878
 353-> 13878 13878 13878 13878 13878 13878 13878 13878
 361-> 13878 13878 13878 13878 13878 13878 13878 13878
 369-> 13878 13878 13878 13878 13878 13878 13878 13878
 377-> 13878 13878 13878 13878 13878 13878 13878 13878
 385-> 13878 13878 13878 13878 13878 13878 13878 13878
 393-> 13878 13878 13878 13878 13878 13878 13878 13878
 401-> 13878 13878 13878 13878 13878 13878 13878 13878
 409-> 13878 13878 13878 13878 13878 13878 13878 13878
 417-> 13878 13878 13878 13878 13878 13878 13878 13878
 425-> 13878 13878 13878 13878 13878 13878 13878 13878
 433-> 13878 13878 13878 13878 13878 13878 13878 13878
 441-> 13878 13878 13878 13878 13878 13878 13878 13878
 449-> 13878 13878 13878 13878 13878 13878 13878 13878
 457-> 13878 13878 13878 13878 13878 13878 13878 13878
 465-> 13878 13878 13878 13878 13878 13878 13878 13878
 473-> 13878 13878 13878 13878 13878 13878 13878 13878
 481-> 13878 13878 13878 13878 13878 13878 13878 13878
 489-> 13878 13878 13878 13878 13878 13878 13878 13878
 497-> 13878 13878 13878 13878 13878 13878 13878 13878
 505-> 13878 13878 13878 13878 13878 13878 13878 13878
[root@dhcp-10-163-102-46 pcie_data_simulator]# $MIDASSYS/bin/mdump
- MIDAS revision: Fri Feb 2 18:49:36 2024 -0500 -  on branch pioneerMidas -- Enter <!> to Exit ------- Midas Dump ---
------------------------ Event# 1 ------------------------
Evid:0002- Mask:0000- Serial:55- Time:0x66e2c435- Dsize:1044/0x414
#banks:1 - Bank list:-CR00-

Bank:CR00 Length: 1024(I*1)/256(I*4)/512(Type) Type:Signed Integer*2
   1-> 14135 14135 14135 14135 14135 14135 14135 14135
   9-> 14135 14135 14135 14135 14135 14135 14135 14135
  17-> 14135 14135 14135 14135 14135 14135 14135 14135
  25-> 14135 14135 14135 14135 14135 14135 14135 14135
  33-> 14135 14135 14135 14135 14135 14135 14135 14135
  41-> 14135 14135 14135 14135 14135 14135 14135 14135
  49-> 14135 14135 14135 14135 14135 14135 14135 14135
  57-> 14135 14135 14135 14135 14135 14135 14135 14135
  65-> 14135 14135 14135 14135 14135 14135 14135 14135
  73-> 14135 14135 14135 14135 14135 14135 14135 14135
  81-> 14135 14135 14135 14135 14135 14135 14135 14135
  89-> 14135 14135 14135 14135 14135 14135 14135 14135
  97-> 14135 14135 14135 14135 14135 14135 14135 14135
 105-> 14135 14135 14135 14135 14135 14135 14135 14135
 113-> 14135 14135 14135 14135 14135 14135 14135 14135
 121-> 14135 14135 14135 14135 14135 14135 14135 14135
 129-> 14135 14135 14135 14135 14135 14135 14135 14135
 137-> 14135 14135 14135 14135 14135 14135 14135 14135
 145-> 14135 14135 14135 14135 14135 14135 14135 14135
 153-> 14135 14135 14135 14135 14135 14135 14135 14135
 161-> 14135 14135 14135 14135 14135 14135 14135 14135
 169-> 14135 14135 14135 14135 14135 14135 14135 14135
 177-> 14135 14135 14135 14135 14135 14135 14135 14135
 185-> 14135 14135 14135 14135 14135 14135 14135 14135
 193-> 14135 14135 14135 14135 14135 14135 14135 14135
 201-> 14135 14135 14135 14135 14135 14135 14135 14135
 209-> 14135 14135 14135 14135 14135 14135 14135 14135
 217-> 14135 14135 14135 14135 14135 14135 14135 14135
 225-> 14135 14135 14135 14135 14135 14135 14135 14135
 233-> 14135 14135 14135 14135 14135 14135 14135 14135
 241-> 14135 14135 14135 14135 14135 14135 14135 14135
 249-> 14135 14135 14135 14135 14135 14135 14135 14135
 257-> 14135 14135 14135 14135 14135 14135 14135 14135
 265-> 14135 14135 14135 14135 14135 14135 14135 14135
 273-> 14135 14135 14135 14135 14135 14135 14135 14135
 281-> 14135 14135 14135 14135 14135 14135 14135 14135
 289-> 14135 14135 14135 14135 14135 14135 14135 14135
 297-> 14135 14135 14135 14135 14135 14135 14135 14135
 305-> 14135 14135 14135 14135 14135 14135 14135 14135
 313-> 14135 14135 14135 14135 14135 14135 14135 14135
 321-> 14135 14135 14135 14135 14135 14135 14135 14135
 329-> 14135 14135 14135 14135 14135 14135 14135 14135
 337-> 14135 14135 14135 14135 14135 14135 14135 14135
 345-> 14135 14135 14135 14135 14135 14135 14135 14135
 353-> 14135 14135 14135 14135 14135 14135 14135 14135
 361-> 14135 14135 14135 14135 14135 14135 14135 14135
 369-> 14135 14135 14135 14135 14135 14135 14135 14135
 377-> 14135 14135 14135 14135 14135 14135 14135 14135
 385-> 14135 14135 14135 14135 14135 14135 14135 14135
 393-> 14135 14135 14135 14135 14135 14135 14135 14135
 401-> 14135 14135 14135 14135 14135 14135 14135 14135
 409-> 14135 14135 14135 14135 14135 14135 14135 14135
 417-> 14135 14135 14135 14135 14135 14135 14135 14135
 425-> 14135 14135 14135 14135 14135 14135 14135 14135
 433-> 14135 14135 14135 14135 14135 14135 14135 14135
 441-> 14135 14135 14135 14135 14135 14135 14135 14135
 449-> 14135 14135 14135 14135 14135 14135 14135 14135
 457-> 14135 14135 14135 14135 14135 14135 14135 14135
 465-> 14135 14135 14135 14135 14135 14135 14135 14135
 473-> 14135 14135 14135 14135 14135 14135 14135 14135
 481-> 14135 14135 14135 14135 14135 14135 14135 14135
 489-> 14135 14135 14135 14135 14135 14135 14135 14135
 497-> 14135 14135 14135 14135 14135 14135 14135 14135
 505-> 14135 14135 14135 14135 14135 14135 14135 14135
[root@dhcp-10-163-102-46 pcie_data_simulator]#

It looks like it's just cycling between 0xFFFFFF00 and 0xFFFFFFFF which is expected (but not intended) behavior in this case.